atunnel: broker actor certificates through atelet by EItanya · Pull Request #708 · agent-substrate/substrate

Eitan Yarmush (EItanya) · 2026-08-03T04:24:11Z

Closes #706

Summary

broker short-lived actor certificates from atelet over a same-node mTLS Unix socket
keep the actor private key in atunnel and renew the certificate before expiry
authenticate egress CONNECT using the actor certificate instead of bearer tokens

Testing

make verify
go test -race ./internal/atunnel

Lior Lieberman (LiorLieberman) · 2026-08-03T16:09:39Z

Thanks Eitan Yarmush (@EItanya) !

Quick question before I review more deeply today.

With the goal of "having a JWT available to atunnel to append on a CONNECT request to the egress" -- What parts of the this PR addresses?

Here are the parts we needed (before this pr -- havent reviewed it yet);

Improve the JWT issuance flow as part of MintJWT (properly authorize atelets, having the JWT formats solid, etc)
Implement an in-memory cache with go background routine that refreshes certs for every active actor.
Implement the UDS socket between atunnel and atelet to pass the certs+jwts.

From a very brief glance - I think your PR is attempting to do all three. Is that correct?

Also, For (1), we likely need to sort out two things to make the JWTs useful:

Get the issuer a publicly accessible DNS name that is also oidc-compliant (cc: Taahir Ahmed (@ahmedtd))
Agree on the format for the Subject field on JWTs

See for ref -

substrate/cmd/ateapi/internal/actoridentity/actoridentity.go

Lines 135 to 138 in 3ed6aa0

    
           // TODO: This is currently API but it has to be a globally unique, oidc-compliant and accsible DNS name 
        
           Issuer: "https://api.ate-system.svc", 
        
           // TODO: this format is very likely going to change. 
        
           Subject:    fmt.Sprintf("atespaces:%s:actors:%s", req.GetAtespace(), req.GetActorName()),

Eitan Yarmush (EItanya) · 2026-08-03T17:13:59Z

Yes, with two clarifications: this PR renews JWTs, not actor certificates, and each worker has only one active actor.

JWT issuance/authorization: implemented. Atunnel authenticates to atelet over mTLS using the worker Pod identity. Atelet derives the worker UID from that certificate and resolves its current actor assignment; the request does not supply actor identity. Ateapi then independently revalidates the atelet’s node, worker assignment, actor UID, and permitted audience before minting. The JWT includes the actor UID/resource version and worker Pod UID.
In-memory renewal: implemented for the worker’s single active actor. Atunnel mints before activation, stores the JWT in memory, renews with roughly 10% lifetime remaining, retries until expiry, and fails closed for new tunnels after expiry. Deactivation clears the JWT and stops renewal.
UDS broker: implemented. The UDS carries only JWT requests/responses—not certificates. The Pod certificate is used to mutually authenticate the atunnel↔atelet connection, including verifying both are on the same node.

I agree the issuer and subject contracts remain unresolved. This PR intentionally retains the existing issuer/subject behavior and does not implement PEP verification yet. Before verification lands, we need a stable OIDC issuer with discovery/JWKS reachable by the PEP.

Lior Lieberman (LiorLieberman) · 2026-08-04T17:38:00Z

+	// TODO: Before release, request an atunnel-specific MintCert purpose and
+	// require the egress PEP to reject generic actor certificates.
+	// The request deliberately carries no actor identity. The authenticated Pod
+	// UID is the only input used to select a worker and its current assignment.
+	workerUID, err := authenticatedWorkerUID(ctx)


not sure I am following the TODO comment. I thought we were saying that we need it to be an actor cert thats specifically for atunnel use.

I think it would be fine to merge without that, and then add it in a follow up? Either way works.

Done. MintCert now requires the ATUNNEL purpose, embeds it in the ActorIdentity certificate extension, and atunnel verifies it before installing the certificate. PEP-side purpose enforcement remains a before-release follow-up.

[needs issue] Eitan Yarmush (@EItanya) do we need an issue for that?

I can create one, but really this should just go straight into your PR

Taahir Ahmed (ahmedtd) · 2026-08-05T01:01:11Z

+	// TODO: Before release, request an atunnel-specific MintCert purpose and
+	// require the egress PEP to reject generic actor certificates.
+	// The request deliberately carries no actor identity. The authenticated Pod
+	// UID is the only input used to select a worker and its current assignment.
+	workerUID, err := authenticatedWorkerUID(ctx)


I think it would be fine to merge without that, and then add it in a follow up? Either way works.

Taahir Ahmed (ahmedtd) · 2026-08-05T01:09:56Z

+		serverboot.Fatal(ctx, "Failed to load atelet Pod identity", fmt.Errorf("credential bundle has no Pod identity"))
+	}
+	brokerTLS := tlsCfg.Clone()
+	brokerTLS.VerifyConnection = restrictClientToNode(ateletIdentity)


I think over a UDS we should be able to get this level of control just from filesystem permissions? Ie, we create a UDS socket file for each individual ateom, and make sure only that ateom can access it.

TLS over a UDS is a bit strange.

Sure, we can do that. To get the same isolation from filesystem permissions, though, we would need a separate broker hostPath, mount only each worker’s own socket directory into its ateom, and manage one listener per worker. The current hostPath is shared by all root-running ateoms, so 0600 on a per-worker filename alone would not isolate them.

mTLS gives us the same worker-to-atelet binding with considerably less change in the current layout. Would you be okay with keeping mTLS here?

[needs issue] lets open an for that

Tracking this investigation in #778: #778

Eitan Yarmush (EItanya) · 2026-08-05T17:03:11Z

Correction to my earlier summary: this PR has been retooled from bearer JWTs to purpose-scoped actor certificates. Atunnel generates and retains a per-activation private key, sends only a CSR to the node-local atelet, and atelet asks ateapi to mint an ATUNNEL-purpose actor certificate after revalidating the exact worker assignment. Atunnel renews that certificate in memory; no actor JWT is transported, so JWT issuer/subject work is no longer in scope here. PEP-side purpose enforcement remains a before-release follow-up.

Taahir Ahmed (ahmedtd) · 2026-08-05T22:04:53Z

LGTM for the certificate bits.

Bowei Du (bowei)

look ok. I think we might want to clean up some of the API comments + make sure optional vs required is what we want.

Lior Lieberman (LiorLieberman) · 2026-08-06T16:46:07Z

LGTM
Will merge once workflows are done.

Eitan Yarmush (@EItanya) please open the issues here (added [needs issue] tag)

Also we want to remove the egressgatewayadress flag and replace it with a proper proto on the actor.

Eitan Yarmush (EItanya) marked this pull request as ready for review August 3, 2026 13:39

Eitan Yarmush (EItanya) requested review from Lior Lieberman (LiorLieberman) and Dmitry Berkovich (dberkov) August 3, 2026 13:39

Eitan Yarmush (EItanya) force-pushed the issue-706-atunnel-identity branch from 3306005 to facebf3 Compare August 4, 2026 01:03

Eitan Yarmush (EItanya) changed the title ~~atunnel: broker actor JWTs through atelet~~ atunnel: broker actor certificates through atelet Aug 4, 2026

Lior Lieberman (LiorLieberman) reviewed Aug 4, 2026

View reviewed changes

Eitan Yarmush (EItanya) mentioned this pull request Aug 4, 2026

Actor Identity Broker can issue an mTLS certificate that identifies a particular atunnel acting on behalf of a particular caller. #734

Open

Taahir Ahmed (ahmedtd) reviewed Aug 5, 2026

View reviewed changes

Max Thompson (thompsonmax) reviewed Aug 5, 2026

View reviewed changes

Comment thread internal/atunnel/egress.go

Eitan Yarmush (EItanya) force-pushed the issue-706-atunnel-identity branch 2 times, most recently from 960bb6d to 03aac19 Compare August 5, 2026 17:01

Max Thompson (thompsonmax) reviewed Aug 5, 2026

View reviewed changes

Comment thread internal/atunnel/egress.go Outdated

Comment thread internal/proto/ateompb/ateom.proto

Comment thread internal/atunnel/egress.go

Comment thread cmd/atelet/credentialbroker.go Outdated

Max Thompson (thompsonmax) reviewed Aug 5, 2026

View reviewed changes

Comment thread internal/atunnel/egress.go

Comment thread internal/atunnel/credential.go

Bowei Du (bowei) reviewed Aug 5, 2026

View reviewed changes

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated

Comment thread internal/proto/ateompb/ateom.proto

Comment thread internal/proto/ateompb/ateom.proto Outdated

Comment thread internal/proto/ateletpb/atelet.proto

This was referenced Aug 6, 2026

egress: add egress gateway support - atenet-egress #693

Open

Investigate replacing credential broker mTLS with per-worker UDS isolation #778

Open

Eitan Yarmush (EItanya) force-pushed the issue-706-atunnel-identity branch from dc595ac to 12511d4 Compare August 7, 2026 14:18

Eitan Yarmush (EItanya) added 4 commits August 7, 2026 14:20

atunnel: broker actor certificates through atelet

d036e8b

atunnel: simplify certificate minting

c5ffcc6

atunnel: address credential review feedback

af350de

ci: retrigger checks

a5caf21

Eitan Yarmush (EItanya) force-pushed the issue-706-atunnel-identity branch from 12511d4 to a5caf21 Compare August 7, 2026 14:22

Lior Lieberman (LiorLieberman) merged commit c9777b4 into agent-substrate:main Aug 7, 2026
13 of 15 checks passed

Eitan Yarmush (EItanya) mentioned this pull request Aug 7, 2026

Configurable JWT authentication to ateapi #757

Open

Eitan Yarmush (EItanya) deleted the issue-706-atunnel-identity branch August 7, 2026 18:46

Uh oh!

Conversation

Eitan Yarmush (EItanya) commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

Lior Lieberman (LiorLieberman) commented Aug 3, 2026

Uh oh!

Eitan Yarmush (EItanya) commented Aug 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Eitan Yarmush (EItanya) commented Aug 5, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Taahir Ahmed (ahmedtd) commented Aug 5, 2026

Uh oh!

Bowei Du (bowei) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lior Lieberman (LiorLieberman) commented Aug 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Eitan Yarmush (EItanya) commented Aug 3, 2026 •

edited

Loading